{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Image Similarity Calculation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "This notebook introduces how to calculate the image similarity with deep learning neural networks provided in Analytics Zoo. A large number of problems in the computer vision domain can be solved by ranking images according to their similarity. For example, \n",
    "1. E-retailers recommend customers products that are similar to items from history purchases, \n",
    "2. Search engine provides ranked result based on the query image provided by users.\n",
    "\n",
    "In this notebook, we use a scenario in real estate industry as an example. Real estate industry has become increasingly digital over the past decade. More than 90 percent of homebuyers search online in the process of seeking a property2. Homeowners and real estate professionals provide information on house characteristics such as location, size, and age, as well as many interior and exterior photos for real estate listing searches. However, due to technical constraints, the enormous amount of information in the photos cannot be extracted and indexed to enhance search or serve real estate listing results. In fact, show me similar homes is a top wish list request among users. By tapping into the available reservoir of image data to power web plus mobile digital experiences, the opportunity to drive greater user satisfaction from improved search relevancy is now a reality. Using Intel’s Analytics Zoo distributed deep learning framework, the recommendation system is designed to play a role in the home buying experience through efficient index and query operations among millions of house images. Users can select a listing photo and have the system recommend listings of similar visual characteristics that may be of interest. The following provides additional parameters to the image similarity search:\n",
    "\n",
    "1. Recommend houses based on title image characteristics and similarity. Most title images are front exterior, while others can be a representative image for the house.\n",
    "2. Low latency API for online querying (< 0.1s)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview of Image Similarity\n",
    "\n",
    "In the research community, image similarity can mean either semantic similarity or visual similarity. Semantic similarity means that both images contain the same category of objects. For example, a ranch house and a traditional house are similar in terms of category (both houses), but may look completely different. Visual similarity, on the other hand, does not care about the object categories but measures how images look like each other from a visual perspective; for example, an apartment image and a traditional house image may be quite similar.\n",
    "\n",
    "Semantic similarity:\n",
    "\n",
    "![using-big-dl-fig1-house1.png](./image/using-big-dl-fig1-house1.png)\n",
    "![using-big-dl-fig1-house1.png](./image/using-big-dl-fig2-house2.png)\n",
    "\n",
    "\n",
    "\n",
    "Visual similarity:\n",
    "![using-big-dl-fig1-house1.png](./image/using-big-dl-fig3-house3.png)\n",
    "![using-big-dl-fig1-house1.png](./image/using-big-dl-fig4-house4.png)\n",
    "\n",
    "For semantic similarity, usually it's an image classification problem, and can be effectively resolved with the popular image perception models like GoogLeNet or ResNet.\n",
    "\n",
    "For visual similarity, there have been many techniques applied across the history:\n",
    "\n",
    "1. SIFT, SURF, color histogram\n",
    "Conventional feature descriptors can be used to compare image similarity. SIFT feature descriptor is invariant to uniform scaling, orientation, and illumination changes, and makes it useful for applications like finding a small image within a larger image.\n",
    "\n",
    "2. pHash\n",
    "This mathematical algorithm analyzes an image's content and represents it using a 64-bit number fingerprint. Two images’ pHash values are close to one another if the images’ content features are similar.\n",
    "\n",
    "3. Image embedding with convolutional neural networks (convnet)\n",
    "Finding the image embedding from the convnet; usually it’s the first linear layer after the convolution and pooling.\n",
    "4. Siamese Network or Deep Ranking\n",
    "A more thorough deep learning solution, but the result model depends heavily on the training data, and may lose generality.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Solution with Analytics Zoo\n",
    "To recommend houses based on image similarity, we first compare the query image of the selected listing photo\n",
    "with the title images of candidate houses. Next, a similarity score for each candidate house is generated.\n",
    "Only the top results are chosen based on ranking. By working with domain experts, the following measure\n",
    "for calculating similarity for house images was developed.\n",
    "```\n",
    "For each image in the candidates, compare with query image {\n",
    " \n",
    "     category score: scene of the image (Bedroom, Kitchen, exterior etc.)\n",
    "      \n",
    "     tag score: Compare important semantic tags. (ranch style, hardwood-floor etc.)\n",
    " \n",
    "     visual score: Visually similarity score, higher is better\n",
    "      \n",
    "     final Score = class score (decisive)   //~1\n",
    "                 + tag score (significant)  //~0.3\n",
    "                 + visual score             //[0,1]\n",
    "}\n",
    "```\n",
    "In this project, both semantic similarity and visual similarity were used. BigDL provides a rich set of functionalities to support training or inference image similarity models, including:\n",
    "\n",
    "-  Providing helpful image readers and transformers based on Apache Spark and OpenCV* for parallel image preprocessing.\n",
    "-  Natively supporting the Spark ML* Estimator/Transformer interface, so that users can perform deep learning training and inference within the Spark ML pipeline.\n",
    "-  Providing convenient model fine-tuning support and a flexible programming interface for model adjustment. Users can load pretrained Caffe*, Torch* or TensorFlow* models into BigDL for fine-tuning or inference.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Semantic similarity Model\n",
    "\n",
    "For semantic similarity, multiple image classification models are used to category the image scene and add import semantic tags to image. Each tag (E.g. House style, number of stories) normally would require an indivisual multiClass classification model.\n",
    "\n",
    "Here we train a scene classification model as a sample. We use training data from http://places2.csail.mit.edu/explore.html to train a multiclassification model, related categories are \n",
    "bathroom, bedroom, house, kitchen.\n",
    "\n",
    "We will fine tune the model from a pretrained Inception on Places2 dataset. \n",
    "\n",
    "1. Download the model from https://github.com/CSAILVision/places365\n",
    "more concretely GoogLeNet-places365: [deploy](https://raw.githubusercontent.com/CSAILVision/places365/master/deploy_googlenet_places365.prototxt) [weights](http://places2.csail.mit.edu/models_places365/googlenet_places365.caffemodel)\n",
    "\n",
    "2. Download the images from http://places2.csail.mit.edu/download.html \n",
    "more concretely [Small images (256 * 256)](http://data.csail.mit.edu/places/places365/train_256_places365standard.tar) Train images. 24GB. MD5: 53ca1c756c3d1e7809517cc47c5561c5.\n",
    "Unpack the downloaded tar file and copy the bathroom, bedroom, house, kitchen folders to a new folder named imageClassification, in which each category resides in its own sub foler. E.g. bedroom images will be in the path \".../imageClassification/bedroom\".\n",
    "    \n",
    "3. Start training with the following code:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "creating: createDefault\n",
      "creating: createSGD\n",
      "creating: createSeqToTensor\n",
      "creating: createSeqToTensor\n",
      "creating: createSeqToTensor\n",
      "creating: createSeqToTensor\n",
      "creating: createSeqToTensor\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "from bigdl.nn.criterion import CrossEntropyCriterion\n",
    "from pyspark.ml import Pipeline\n",
    "from pyspark.sql.functions import col, udf\n",
    "from pyspark.sql.types import DoubleType, StringType\n",
    "\n",
    "from zoo.common.nncontext import *\n",
    "from zoo.feature.image import *\n",
    "from zoo.pipeline.api.keras.layers import Dense, Input, Flatten\n",
    "from zoo.pipeline.api.net import *\n",
    "from zoo.pipeline.nnframes import *\n",
    "\n",
    "sc = init_nncontext(\"ImageSimilarityExample\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training data distribution:\n",
      "+-----+-----+\n",
      "|label|count|\n",
      "+-----+-----+\n",
      "|  1.0| 2009|\n",
      "|  4.0| 1968|\n",
      "|  3.0| 2004|\n",
      "|  2.0| 2050|\n",
      "+-----+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "image_path = \"/tmp/images/imageClassification/*/*\"\n",
    "imageDF = NNImageReader.readImages(image_path, sc, 24).sample(False, 0.4)\n",
    "\n",
    "def getLabel(path):\n",
    "    if \"bathroom\" in path: label = 1.0\n",
    "    elif \"bedroom\" in path: label = 2.0\n",
    "    elif \"house\" in path: label = 3.0\n",
    "    else: label = 4.0\n",
    "    return label\n",
    "\n",
    "getLabelUDF = udf(lambda row: getLabel(row[0]), DoubleType())\n",
    "labelDF = imageDF.withColumn(\"label\", getLabelUDF(col('image')))\n",
    "(trainingDF, validationDF) = labelDF.randomSplit([0.8, 0.2])\n",
    "trainingDF.cache()\n",
    "print(\"training data distribution:\")\n",
    "trainingDF.groupBy(\"label\").count().show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model_def_path = \"/tmp/images/googlenet_places365/deploy.prototxt\"\n",
    "model_weight_path = \"/tmp/images/googlenet_places365/googlenet_places365.caffemodel\"\n",
    "\n",
    "# load pretrained caffe model\n",
    "preTrained_model = Net.load_caffe(model_def_path, model_weight_path)\n",
    "\n",
    "# create a new model by remove layers after pool5/drop_7x7_s1\n",
    "part_model = preTrained_model.new_graph([\"pool5/drop_7x7_s1\"])\n",
    "\n",
    "# optionally freeze layers from input to pool4/3x3_s2 inclusive\n",
    "# model.freeze_up_to([\"pool4/3x3_s2\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "creating: createSequential\n",
      "creating: createView\n",
      "creating: createLinear\n",
      "creating: createLogSoftMax\n",
      "creating: createRowToImageFeature\n",
      "creating: createImageResize\n",
      "creating: createImageCenterCrop\n",
      "creating: createImageChannelNormalize\n",
      "creating: createImageMatToTensor\n",
      "creating: createImageFeatureToTensor\n",
      "creating: createChainedPreprocessing\n",
      "creating: createCrossEntropyCriterion\n",
      "creating: createScalarToTensor\n",
      "creating: createFeatureLabelPreprocessing\n",
      "creating: createNNClassifier\n",
      "creating: createToTuple\n",
      "creating: createChainedPreprocessing\n",
      "creating: createTensorToSample\n",
      "creating: createChainedPreprocessing\n",
      "Finished training\n"
     ]
    }
   ],
   "source": [
    "from bigdl.nn.layer import Sequential, Linear, View, LogSoftMax\n",
    "from bigdl.nn.criterion import CrossEntropyCriterion\n",
    "\n",
    "# add a new linear layer with numClass outputs, in our example, it's 6.\n",
    "scene_network = Sequential().add(part_model).add(View([1024])).add(Linear(1024, 6)).add(LogSoftMax())\n",
    "\n",
    "transformer = ChainedPreprocessing(\n",
    "    [RowToImageFeature(), ImageResize(256, 256), ImageCenterCrop(224, 224),\n",
    "     ImageChannelNormalize(123.0, 117.0, 104.0), ImageMatToTensor(), ImageFeatureToTensor()])\n",
    "\n",
    "classifier = NNClassifier(scene_network, CrossEntropyCriterion(), transformer) \\\n",
    "    .setLearningRate(0.001).setLearningRateDecay(1e-3) \\\n",
    "    .setBatchSize(20).setMaxEpoch(2).setFeaturesCol(\"image\") \\\n",
    "    .setCachingSample(False)\n",
    "\n",
    "# train the model, training progress is available in Jupyter notebook console.\n",
    "scene_classification_model = classifier.fit(trainingDF)\n",
    "print(\"Finished training\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start evaluation:\n",
      "+--------------------+-----+----------+\n",
      "|               image|label|prediction|\n",
      "+--------------------+-----+----------+\n",
      "|[file:/tmp/images...|  2.0|       2.0|\n",
      "|[file:/tmp/images...|  2.0|       2.0|\n",
      "|[file:/tmp/images...|  2.0|       2.0|\n",
      "|[file:/tmp/images...|  2.0|       2.0|\n",
      "|[file:/tmp/images...|  3.0|       3.0|\n",
      "|[file:/tmp/images...|  3.0|       3.0|\n",
      "|[file:/tmp/images...|  3.0|       3.0|\n",
      "|[file:/tmp/images...|  3.0|       3.0|\n",
      "|[file:/tmp/images...|  1.0|       1.0|\n",
      "|[file:/tmp/images...|  1.0|       1.0|\n",
      "|[file:/tmp/images...|  1.0|       1.0|\n",
      "|[file:/tmp/images...|  1.0|       1.0|\n",
      "|[file:/tmp/images...|  1.0|       1.0|\n",
      "|[file:/tmp/images...|  4.0|       4.0|\n",
      "+--------------------+-----+----------+\n",
      "\n",
      "Accuracy = 0.974333 \n"
     ]
    }
   ],
   "source": [
    "# evaluate the model\n",
    "print(\"Start evaluation:\")\n",
    "predictionDF = scene_classification_model.transform(validationDF).cache()\n",
    "predictionDF.sample(False, 0.01).show()\n",
    "\n",
    "correct = predictionDF.filter(\"label=prediction\").count()\n",
    "overall = predictionDF.count()\n",
    "accuracy = correct * 1.0 / overall\n",
    "predictionDF.unpersist()\n",
    "trainingDF.unpersist()\n",
    "print(\"Accuracy = %g \" % accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visual Similarity Model\n",
    "\n",
    "We first built an evaluation dataset to choose the best options for image similarity computation. In the evaluation dataset, each record contains three images.\n",
    "\n",
    "```Triplet (query image, positive image, negative image)```\n",
    "\n",
    "where positive image is more similar to the query image than the negative image\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the four methods listed above for computing visually similarity, Siamese Network or Deep Ranking appear to be more precise, but due to the lack of training data to support meaningful models, the results were inconclusive. With the help of the evaluation dataset we tried the other three methods, and both SIFT and pHash produced unreasonable results. We suspect that was because both of them cannot represent the essential characteristics of real estate images.\n",
    "\n",
    "Using image embedding from the pretrained deep learning models on the Places dataset, the expected precision accuracy level was achieved:\n",
    "\n",
    "Network | Embedding  | Precision\n",
    "------- | -------- | --------\n",
    "Deepbit*| 1024 binary |80%\n",
    "GoogLeNet*|1024 floats |84%\n",
    "VGG-16 | 25088 floats | 93%"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarity (m1, m2) = cosine (embedding (m1), embedding (m2)).\n",
    "\n",
    "After L2 normalization, cosine similarity can be computed very efficiently. While VGG-16 embedding has a clear advantage, we also tried the SVM model trained from the evaluation dataset to assign different weight to each of the embedding features, but this only gives limited improvement, and we are concerned that the SVM model may not be general enough to cover the real-world images.\n",
    "\n",
    "Next we demo how to extract image embedding with a pretrained VGG network on places2 dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A pretrained VGG network can be downloaded from https://github.com/CSAILVision/places365, more concretely:\n",
    "VGG16-places365: [deploy](https://raw.githubusercontent.com/CSAILVision/places365/master/deploy_vgg16_places365.prototxt) [weights](http://places2.csail.mit.edu/models_places365/vgg16_places365.caffemodel)\n",
    "\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "creating: createRowToImageFeature\n",
      "creating: createImageResize\n",
      "creating: createImageCenterCrop\n",
      "creating: createImageChannelNormalize\n",
      "creating: createImageMatToTensor\n",
      "creating: createImageFeatureToTensor\n",
      "creating: createChainedPreprocessing\n"
     ]
    }
   ],
   "source": [
    "\n",
    "vgg_def_path = \"/tmp/images/vgg_16_places365/deploy.prototxt\"\n",
    "vgg_weight_path = \"/tmp/images/vgg_16_places365/vgg16_places365.caffemodel\"\n",
    "\n",
    "transformer = ChainedPreprocessing(\n",
    "    [RowToImageFeature(), ImageResize(256, 256), ImageCenterCrop(224, 224),\n",
    "     ImageChannelNormalize(123.0, 117.0, 104.0), ImageMatToTensor(), ImageFeatureToTensor()])\n",
    "\n",
    "full_model = Net.load_caffe(vgg_def_path, vgg_weight_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "creating: createSequential\n",
      "creating: createView\n",
      "creating: createNormalize\n",
      "creating: createTensorToSample\n",
      "creating: createChainedPreprocessing\n",
      "creating: createNNModel\n",
      "+--------------------+--------------------+\n",
      "|               image|          prediction|\n",
      "+--------------------+--------------------+\n",
      "|[file:/tmp/images...|[0.0, 0.0, 0.0, 0...|\n",
      "|[file:/tmp/images...|[0.0, 0.0, 0.0, 0...|\n",
      "|[file:/tmp/images...|[0.0, 0.0, 0.0, 0...|\n",
      "|[file:/tmp/images...|[0.0, 0.0, 0.0, 0...|\n",
      "|[file:/tmp/images...|[0.0, 0.0, 0.0048...|\n",
      "|[file:/tmp/images...|[0.0, 0.0, 0.0, 0...|\n",
      "|[file:/tmp/images...|[0.0, 0.0, 0.0, 0...|\n",
      "|[file:/tmp/images...|[0.0, 0.0, 0.0, 0...|\n",
      "|[file:/tmp/images...|[0.0, 0.0, 0.0, 0...|\n",
      "|[file:/tmp/images...|[0.0, 0.0, 0.0, 0...|\n",
      "+--------------------+--------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from bigdl.nn.layer import Sequential, View, Normalize\n",
    "\n",
    "# create a new model by removing layers after pool5\n",
    "model = full_model.new_graph([\"pool5\"])\n",
    "# generates a vector of dimension 25088 for each image\n",
    "vggModel = Sequential().add(model).add(View([25088])).add(Normalize(2.0))\n",
    "\n",
    "embeddingModel = NNModel(vggModel, transformer).setFeaturesCol(\"image\")\n",
    "embeddingModel.transform(imageDF.limit(10)).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Qury for similar images:\n",
    "\n",
    "After getting the sample scene classification model and VGG model, we can process all the images to extract category info and embedding. The pre-calculated information helps for the online API query and meet the low latecy requirement.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "16 images cached in imageDict\n"
     ]
    }
   ],
   "source": [
    "# we selected a combination of sample images from places dataset in the samples folder\n",
    "image_path = \"./samples\"\n",
    "imageDF = NNImageReader.readImages(image_path, sc)\n",
    "\n",
    "pathDF = with_origin_column(imageDF)\n",
    "sceneDF = scene_classification_model.setBatchSize(4).setPredictionCol(\"sceneClassification\").transform(pathDF)\n",
    "embeddingDF = embeddingModel.setBatchSize(4).setPredictionCol(\"embedding\").transform(sceneDF)\n",
    "\n",
    "# save data in a dict, where key is the path of the image, value are category and embedding\n",
    "imageDict = embeddingDF.select(\"origin\", \"sceneClassification\", \"embedding\").rdd \\\n",
    "    .map(lambda r: (r[0], (r[1], r[2]))) \\\n",
    "    .collectAsMap()\n",
    "print(str(len(imageDict)) + \" images cached in imageDict\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We just picked a random house image as the query image, E.g. house/00000099.jpg\n",
    " ![00000099.jpg](./samples/00000099.jpg)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['file:./samples/00000099.jpg', 'file:./samples/00003201.jpg', 'file:./samples/00000104.jpg']\n"
     ]
    }
   ],
   "source": [
    "from pyspark.sql.types import DoubleType\n",
    "from pyspark.sql.functions import udf, struct, col\n",
    "import numpy as np\n",
    "from heapq import nlargest\n",
    "\n",
    "query_origin = \"file:./samples/00000099.jpg\"\n",
    "query_record = imageDict[query_origin]\n",
    "query_classification = query_record[0]\n",
    "query_embedding = query_record[1]\n",
    "\n",
    "def get_score(x):\n",
    "    candidate_classification = x[0]\n",
    "    candidate_embedding = x[1]\n",
    "    classScore = 1.0 if query_classification == candidate_classification else 0.0\n",
    "    visualScore = np.dot(query_embedding, candidate_embedding)\n",
    "    return classScore + visualScore\n",
    "\n",
    "score_dict = {k: get_score(v) for k, v in imageDict.items()}\n",
    "\n",
    "# select the top 3 most similar images\n",
    "top_3 = nlargest(3, score_dict, key=score_dict.get)\n",
    "print(top_3)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Besides the query image itself, the other two most similar images are ![00000104.jpg](./samples/00000104.jpg) and ![00003201.jpg](./samples/00003201.jpg)\n",
    "\n",
    "In the demo we used Dict to store the inferenced category and embedding for image, alternatively user may use database or other key-value store to cache the result for online query."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Production deployment\n",
    "In production, the project can be separated into three parts:\n",
    "\n",
    "1. Model training (offline)\n",
    "The model training mainly refers to the semantic models (GoogLeNet v1 fine-tuned on the Place dataset) and also finding the proper embedding for visual similarity calculation. Retraining may happen periodically depending on model performance or requirement changes.\n",
    "2. Image inference (online)\n",
    "With the trained semantic models (GoogLeNet v1) in the first step and the pretrained VGG-16, we can convert the images to tags and embeddings, and save the results in a key-value cache. (Apache HBase* or SQL* can also be used).\n",
    "![using-big-dl-fig7-image-processing.png](./image/using-big-dl-fig7-image-processing.png)\n",
    "All the existing images and new images need to go through the inference above and converted into a table structure, as shown:\n",
    "![using-big-dl-fig8-data-table_0.png](./image/using-big-dl-fig8-data-table_0.png)\n",
    "The inference process can happen periodically (for example, one day) or triggered by a new image upload from a real estate listing entry. Each production image only needs to go through the inference process once. With the indexed image tagging and similarity feature, fast query performance is supported in a high concurrency environment.\n",
    "3. API serving for query (online)\n",
    "The house recommendation system exposes a service API to its upstream users. Each query sends a query image and candidate images as parameters. With the indexed image information shown in the table above, we can quickly finish the one-versus- many query. For cosine similarity, processing is very efficient and scalable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Demo\n",
    "\n",
    "We provided One examples from the online website:\n",
    "\n",
    "Example 1\n",
    "![using-big-dl-fig9-home-example1.png](./image/using-big-dl-fig9-home-example1.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "This notebook described how to use Deep Learning neural networks to extract important semantic tags and calculate the image similarity utilizing Intel Analytics Zoo. One deep learning classification models was fine-tuned from pretrained Caffe models in order to extract the important semantic tags from real estate images. We further compared different visual similarity computation methods and found image embedding from VGG to be the most helpful inference model in our case. As an end-to-end industry example, we demonstrated how to leverage deep learning with Analytics Zoo to enable greater deep learning-based image recognition innovation for the industry."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
